US Medical Licensing Exam Performance and American Board of Surgery Qualifying and Certifying Examinations.

BACKGROUND: Program directors use US Medical Licensing Exam (USMLE) scores as criteria for ranking applicants. First-time pass rates of the American Board of Surgery (ABS) Qualifying (QE) and Certifying (CE) Examinations have become important measures of residency program quality. USMLE Step 1 will become pass/fail in 2022. STUDY DESIGN: American Board of Surgery QE and CE success rates were assessed considering multiple characteristics of highly ranked (top 20) applicants to 22 general surgery programs in 2011. Chi-square, t-test, Wilcoxon Rank sum, linear and logistic regression were used, as appropriate. RESULTS: The QE and CE first attempt pass rates were 96% (235/244) and 86% (190/221), respectively. QE/CE success was not significantly associated with sex, race, research experience, or publications. Alpha Omega Alpha (AΩA) status was associated with success on the index CE (98% vs 83%; p = 0.008). Step 1 and Step 2 Clinical Knowledge (CK) scores of surgeons who passed QE on their first attempt were higher than scores of those who failed (Step 1: 233 vs 218; p = 0.016); (Step 2CK: 244 vs 228, p = 0.009). For every 10-point increase in Step 1 and 2CK scores, the odds of passing CE on the first attempt increased 1.5 times (95% CI 1.12, 2.0; p = 0.006) and 1.5 times (95% CI 1.11, 2.02, p = 0.008), respectively. For every 10-point increase in Steps 1 and 2CK scores, the odds of passing the QE on the first attempt increased 1.85 times (95% CI 1.11, 3.09; p = 0.018) and 1.86 times (95% CI 1.14, 3.06, p = 0.013), respectively. CONCLUSIONS: USMLE Step 1 and Step 2 CK examination scores correlate with American Board of Surgery QE and CE performance and success. The USMLE decision to transition Step 1 to a pass/fail examination will require program directors to identify other factors that predict ABS performance for ranking applicants.

Multiple United States Medical Licensing Examination Attempts and the Estimated Risk of Disciplinary Actions Among Graduates of U.S. and Canadian Medical Schools.

PURPOSE: The United States Medical Licensing Examination (USMLE) recently announced 2 policy changes: shifting from numeric score reporting on the Step 1 examination to pass/fail reporting and limiting examinees to 4 attempts for each Step component. In light of these policies, exam measures other than scores, such as the number of examination attempts, are of interest. Attempt limit policies are intended to ensure minimum standards of physician competency, yet little research has explored how Step attempts relate to physician practice outcomes. This study examined the relationship between USMLE attempts and the likelihood of receiving disciplinary actions from state medical boards. METHOD: The sample population was 219,018 graduates from U.S. and Canadian MD-granting medical schools who passed all USMLE Step examinations by 2011 and obtained a medical license in the United States, using data from the NBME and the Federation of State Medical Boards. Logistic regressions estimated how attempts on Steps 1, 2 Clinical Knowledge (CK), and 3 examinations influenced the likelihood of receiving disciplinary actions by 2018, while accounting for physician characteristics. RESULTS: A total of 3,399 physicians (2%) received at least 1 disciplinary action. Additional attempts needed to pass Steps 1, 2 CK, and 3 were associated with an increased likelihood of receiving disciplinary actions (odds ratio [OR]: 1.07, 95% confidence interval [CI]: 1.01, 1.13; OR: 1.09, 95% CI: 1.03, 1.16; OR: 1.11, 95% CI: 1.04, 1.17, respectively), after accounting for other factors. CONCLUSIONS: Physicians who took multiple attempts to pass Steps 1, 2 CK, and 3 were associated with higher estimated likelihood of receiving disciplinary actions. This study offers support for licensure and practice standards to account for physicians' USMLE attempts. The relatively small effect sizes, however, caution policy makers from placing sole emphasis on this relationship.

Admissions Is Not Enough: The Racial Achievement Gap in Medical Education.

The achievement gap is a disparity in academic and standardized test performance that exists between White and underrepresented minority (URM) students that begins as early as preschool and worsens as students progress through the educational system. Medical education is not immune to this inequality. URM medical students are more likely to experience delayed graduation and course failure, even after accounting for science grade point average and Medical College Admission Test performance. Moreover, URM students are more likely to earn lower scores on licensing examinations, which can have a significant impact on their career trajectory, including specialty choice and residency competitiveness. After the release of preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS) and public commentary on these recommendations, the National Board of Medical Examiners and Federation of State Medical Boards announced that the United States Medical Licensing Examination (USMLE) Step 1 would transition from a 3-digit numeric score to pass/fail scoring. Given that another of InCUS's recommendations was to "minimize racial demographic differences that exist in USMLE performance," it is paramount to consider the impact of this scoring change on URM medical students specifically. Holistic admissions are a step in the right direction of acknowledging that URM students often travel a further distance to reach medical school. However, when residency programs emphasize USMLE performance (or any standardized test score) despite persistent test score gaps, medical education contributes to the disproportionate harm URM students face and bolsters segregation across medical specialties. This Perspective provides a brief explanation of the achievement gap, its psychological consequences, and its consequences in medical education; discusses the potential effect of the Step 1 scoring change on URM medical students; and provides a review of strategies to redress this disparity.

Are USMLE Scores Valid Measures for Chief Resident Selection?

BACKGROUND: The US Medical Licensing Examination (USMLE) Step 1 and Step 2 scores are often used to inform a variety of secondary medical career decisions, such as residency selection, despite the lack of validity evidence supporting their use in these contexts. OBJECTIVE: We compared USMLE scores between non-chief residents (non-CRs) and chief residents (CRs), selected based on performance during training, at a US academic medical center that sponsors a variety of graduate medical education programs. METHODS: This was a retrospective cohort study of residents' USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores from 2015 to 2020. The authors used archived data to compare USMLE Step 1 and Step 2 CK scores between non-CR residents in each of the eligible programs and their CRs during the 6-year study period. RESULTS: Thirteen programs enrolled a total of 1334 non-CRs and 211 CRs over the study period. There were no significant differences overall between non-CRs and CRs average USMLE Step 1 (239.81 ± 14.35 versus 240.86 ± 14.31; P = .32) or Step 2 scores (251.06 ± 13.80 versus 252.51 ± 14.21; P = .16). CONCLUSIONS: There was no link between USMLE Step 1 and Step 2 CK scores and CR selection across multiple clinical specialties over a 6-year period. Reliance on USMLE Step 1 and 2 scores to predict success in residency as measured by CR selection is not recommended.

How is increased selectivity of medical school admissions associated with physicians' career choice? A Japanese experience.

BACKGROUND: During the long-lasting economic stagnation, the popularity of medical school has dramatically increased among pre-medical students in Japan. This is primarily due to the belief that medicine is generally a recession-proof career. As a result, pre-medical students today who want to enter medical school have to pass a more rigorous entrance examination than that in the 1980s. This paper explores the association between the selectivity of medical school admissions and graduates' later career choices. METHODS: A unique continuous measure of the selectivity of medical school admissions from 1980 to 2017, which is defined as the deviation value of medical schools, was merged with cross-sectional data of 122 990 physicians aged 35 to 55 years. The association between the deviation value of medical schools and various measures of physicians' career choices was explored by logistic and ordinary least square regression models. Graduates from medical schools in which the deviation value was less than 55 were compared with those from more competitive medical schools, after controlling for fixed effects for the medical school attended by binary variables. RESULTS: From 1980 to 2017, the average deviation value increased from 58.3 to 66.3, indicating a large increase in admission selectivity. Empirical results suggest that increasing selectivity of a medical school is associated with graduates having a higher probability of choosing a career in an acute hospital as well as having a lower probability of opening their own clinic and choosing a career in primary health care. Graduating from a highly competitive medical school (i.e., deviation value of more than 65) significantly increases the probability of working at typical acute hospitals such as so-called 7:1 hospitals (OR 1.665 2, 95%CI 1.444 0-1.920 4) and decreases the probability of working at primary care facilities (OR 0.602 6, 95%CI 0.441 2-0.823 0). It is also associated with graduates having a higher probability of becoming medical board certified (OR 1.294 6, 95%CI 1.108 8-1.511 4). CONCLUSION: Overall, this paper concludes that increased selectivity of medical school admissions predicts a higher quality of physicians in their own specialty, but at the same time, it is associated with a lower supply of physicians who go into primary care.

Overemphasis of Step 1 Scores May Affect Application Pool Diversity in Radiation Oncology.

PURPOSE: Many radiation oncology programs use Step 1 score metrics as a surrogate for intelligence and success to screen applicants. The impact of this practice on radiation oncology applicant pool diversity is unknown. METHODS AND MATERIALS: Electronic Residency Application Service applications submitted to our institution between 2015 and 2018 match cycles were reviewed. Sex, age, race/ethnicity, and Step 1 scores were collected. Groupings by characteristics were sex (female vs male), age ≤30 versus >30 years, and race/ethnicity by underrepresented minority (URM) versus non-URM status. URMs were defined as Black/African American, Hispanic, Native American/Alaskan Native, and Hawaiian/Pacific Islander. Step 1 scores were divided based on scores of 220 and 240. The association between applicant demographics and Step 1 scores was assessed using proportional odds logistic regression for ordinal outcomes. RESULTS: Eight hundred ten applicants with Step 1 scores ranging from 188 to 275 were collected, representing nearly 90% of all applicants during the 2015 to 2018 Electronic Residency Application Service cycles. Twenty-nine percent were female, 29% were >30 years of age, and 10% were URMs. Increasing Step 1 score requirements disproportionately decreased representation of applicants who were female versus male at 240 (-51% vs -31%), >30 versus ≤30 years old at 220 (-28% vs -6%) and 240 (-55% vs -26%), and URMs versus non-URMs at 220 (-34% vs -9%) and 240 (-61% vs -34%). On analysis, Step 1 score requirements had a statistically significantly impact on applicant distributions based on sex, age, and URM status (P < .001). CONCLUSIONS: Overemphasis of Step 1 scores may reduce the diversity of the radiation oncology applicant pool. Further evaluation of practices that counter the stated American Society of Clinical Oncology, American Society of Radiation Oncology, and American College of Radiology diversity missions should be pursued to improve understanding of barriers and biases.

Effects of Moving the United States Medical Licensing Examination Step 1 After Core Clerkships on Step 2 Clinical Knowledge Performance.

PURPOSE: To investigate the effect of a change in the United States Medical Licensing Examination Step 1 timing on Step 2 Clinical Knowledge (CK) scores, the effect of lag time on Step 2 CK performance, and the relationship of incoming Medical College Admission Test (MCAT) score to Step 2 CK performance pre and post change. METHOD: Four schools that moved Step 1 after core clerkships between academic years 2008-2009 and 2017-2018 were analyzed. Standard t tests were used to examine the change in Step 2 CK scores pre and post change. Tests of differences in proportions were used to evaluate whether Step 2 CK failure rates differed between curricular change groups. Linear regressions were used to examine the relationships between Step 2 CK performance, lag time and incoming MCAT score, and curricular change group. RESULTS: Step 2 CK performance did not change significantly (P = .20). Failure rates remained highly consistent (pre change: 1.83%; post change: 1.79%). The regression indicated that lag time had a significant effect on Step 2 CK performance, with scores declining with increasing lag time, with small but significant interaction effects between MCAT and Step 2 CK scores. Students with lower incoming MCAT scores tended to perform better on Step 2 CK when Step 1 was after clerkships. CONCLUSIONS: Moving Step 1 after core clerkships appears to have had no significant impact on Step 2 CK scores or failure rates, supporting the argument that such a change is noninferior to the traditional model. Students with lower MCAT scores benefit most from the change.

The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study.

PURPOSE: The new Medical College Admission Test (MCAT) was introduced in April 2015. This report presents findings from the first study of the validity of scores from the new MCAT exam in predicting student performance in the first year of medical school (M1). METHOD: The authors analyzed data from the national population of 2016 matriculants with scores from the new MCAT exam (N = 7,970) and the sample of 2016 matriculants (N = 955) from 16 medical schools who volunteered to participate in the validity research. They examined correlations of students' MCAT total scores and total undergraduate grade point averages (UGPAs), alone and together, with their summative performance in M1, and the success rate of students with different MCAT scores in their on-time progression to the second year of medical school (M2). They assessed whether MCAT scores provided comparable prediction of performance in M1 by students' race/ethnicity, socioeconomic background, and gender. RESULTS: Correlations of MCAT scores with summative performance in M1 ranged from medium to large. Although MCAT scores and UGPAs provided similar prediction of performance in M1, using both metrics provided better prediction than either alone. Additionally, students with a wide range of MCAT scores progressed to M2 on time. Finally, MCAT scores provided comparable prediction of performance in M1 for students from different sociodemographic backgrounds. CONCLUSIONS: This study provides early evidence that scores from the new MCAT exam predict student performance in M1. Future research will examine the validity of MCAT scores in predicting performance in later years.

USMLE Scores and Clinical Rotation Role in Predicting ABSITE Performance Among Surgery Interns.

BACKGROUND: The American Board of Surgery In-training Examination (ABSITE) is an important marker of medical knowledge. It is unclear what factors predict or improve these scores. We evaluated demographics, United States Medical Licensing Examination (USMLE) step 1 and 2 scores, and surgical rotations during the intern year to determine if there were any correlations with the ABSITE performance. METHODS: This was a multicenter retrospective review during a 6-y period, investigating the correlation and association of demographics, USMLE scores, and types of rotations on the ABSITE percentile score of interns. Demographics included age, gender, race/ethnicity, U.S. versus international/foreign medical graduate for stratified analyses. Descriptive analysis was performed with ANOVA, correlation was evaluated with 95% confidence interval, and significance was defined as P < 0.05. RESULTS: Complete records obtained on 89 interns from six different general surgery programs over 6 y revealed that there was a significant correlation between USMLE 1 and 2 with the ABSITE. USMLE 2 correlation was the strongest (r = 0.44, 95% confidence interval = [0.25-0.60], P < 0.05). There was a significant difference in ABSITE performance (mean score difference of 17.3 percentile, P = 0.01) of interns who had an intensive care unit rotation before examination. Other surgical rotations were not associated with an ABSITE difference. Demographic factors such as age, gender, race/ethnicity, or medical graduate background status were not associated with ABSITE scores. CONCLUSIONS: USMLE step 2 scores had a higher correlation with intern ABSITE performance. An intensive care unit rotation before taking the ABSITE was associated with a significant difference in their percentile scores. Demographic factors were not correlated with ABSITE performance.

Use of the United States Medical Licensing Examination Step-1 Score as a Screening Tool for Orthopaedic Surgery Away Rotations.

BACKGROUND: Orthopaedic surgery has become an increasingly competitive specialty for medical students applying to residency. Aside from residency interviews, away rotations are one of the only opportunities for residency programs to qualitatively evaluate an applicant. The purpose of this study was to determine if residency program directors (PDs) use a minimum United States Medical Licensing Examination (USMLE) Step-1 score as a screening tool for students applying for away rotations at their institution. METHODS: An anonymous 12-question survey about residency selection criteria and which criteria are used to select applicants for away rotations was distributed to PDs at U.S. allopathic orthopaedic surgery residency programs who are members of the Council of Orthopaedic Residency Directors (CORD). Questions included information on minimum USMLE Step-1 scores, the number of students who complete away rotations at the program each year, and how applicants are selected for rotations. RESULTS: Survey responses were received from 87 (58%) of 149 PDs. Seventy-one (82%) of these PDs reported that their program uses a minimum Step-1 score for residency applications, with the most common cutoff score being between 231 and 240 (as answered by 33 [46%] of 72 PDs). Twenty-seven (31%) of 87 PDs reported that if a student does not meet their residency minimum Step-1 score, he or she is not offered the opportunity to rotate at the program. Eleven (25%) of 44 PDs reported that they do not inform students when their score is below the program's minimum. CONCLUSIONS: A substantial proportion of orthopaedic surgery residency PDs use the USMLE Step-1 score as a screening tool for students interested in doing away rotations at their programs. If a student's Step-1 score does not meet the program's minimum, the majority of programs still will allow the student to rotate at their institution. Students should take this information into account when selecting away rotations in order to maximize their chances of matching into an orthopaedic surgery program.